CDRIVER-6071 replace build-and-test-with-toolchain with openssl-compat #2082

eramongodb · 2025-08-06T17:25:28Z

Resolves CDRIVER-6071. Removes dependency on the legacy C toolchain project by replacing the build-and-test-with-toolchain task (which began inexplicably failing since 2025-07-17; did the artifact expire?) with a new openssl-compat matrix.

Historically, the legacy C toolchain project was meant to test both SSL library compatibility and build tool minimum requirements, as currently documented by its README:

This project exists to provide a stable set of tools and libraries at set minimum versions.

However, its purpose has long been made obsolete or superseded by better alternatives. Support for LibreSSL was dropped in CDRIVER-5693. Boost is irrelevant (C++ Driver doesn't support it since CXX-2797, nor does it use the legacy C toolchain anyways). The minimum required CMake version was raised to 3.15 in CDRIVER-4637 (a new branch as created by https://github.com/10gen/mongo-c-toolchain/pull/2 to support this new requirement); this can easily be replaced with a cmake-compat matrix (+ soon powered by uvx cmake instead of find-cmake-latest.sh). The legacy C toolchain also provides zstd, but AFAICT we do not actually test this minimum zstd compatibility requirement, or any other external library dependency's minimum version requirements (we can consider adding additional tasks to accomplish this if needed).

Therefore, the only value (that was) being provided by the build-and-test-with-toolchain task is OpenSSL library compatibility checks, which is worth preserving given the wide variety of OpenSSL library versions across platforms and distros. This PR proposes preserving and refining this compatibility coverage as the new openssl-compat matrix.

First, it is important to note, per OpenSSL docs:

All versions not present above are out of support and should not be used. Users of those versions are encouraged to upgrade to the newest series as soon as possible. Extended support for 1.1.1 and 1.0.2 to gain access to security fixes for those versions is available.

The versions "present above" are currently 3.0 through 3.5. This means we can/should drop compatibility coverage for any version older than 3.0 that is not either 1.0.2 and 1.1.1. Relative to the legacy C toolchain, this means dropping 0.9.8, 1.0.1, and 1.1.0.

Note

The results of an EVG distro survey on 2025-08-05 report the following distros which are currently used by the C Driver that do not have provide OpenSSL v3 or newer via default system packages:

amazon2-arm64-latest: 1.0.2k FIPS
debian-11: 1.1.1n
rhel7-latest: 1.0.2k FIPS
rhel8-latest: 1.1.1k FIPS
rhel8-power: 1.1.1k FIPS
rhel8-zseries: 1.1.1k FIPS
rhel80: 1.1.1 FIPS
rhel84: 1.1.1g FIPS
rhel94: 1.1.1g FIPS
ubuntu2004: 1.1.1f
ubuntu2004-arm64: 1.1.1f

These results support the argument for compatibility-checking pre-v3 OpenSSL versions 1.0.2 and 1.1.1 only.

Furthermore, per docs:

The following OpenSSL version(s) are FIPS validated:

3.1.2: FIPS 140-3

3.0.9: FIPS 140-2

3.0.8: FIPS 140-2

3.0.0: FIPS 140-2

Other OpenSSL Releases MAY use the validated FIPS provider, but MUST NOT build and use their own FIPS provider.

Given we have plenty of OpenSSL coverage with other compile-and-test matrices, I believe we do not need to extend compatibility checks for pre-v3 OpenSSL versions more than we already have. That being said, all "special" OpenSSL-related tasks have been relocated from their prior build variants into the openssl-compat matrix: all remaining -openssl tasks simply use the system-provided OpenSSL library as-is without any special steps.

That being said, even though some system-provided OpenSSL libraries are FIPS-validated, this doesn't necessarily mean the FIPS module is enabled and used by default. Therefore, the openssl-compat matrix includes a few custom OpenSSL installations that do set the appropriate openssl.cnf properties to use the FIPS module by default. (That being said, does mongoc even use the FIPS module programmatically at all?) For now, only 3.0.9 and 3.1.2 are used for FIPS 140-2 and FIPS 140-3 coverage respectively (each being the latest patch release for its minor version series).

Important

This PR deliberately drops OpenSSL FIPS Object Module 2.0 integration checks. OpenSSL 3.0.9 already provides FIPS 140-2 compliance coverage. Per Wikipedia, "The FIPS Object Module 2.0 remained FIPS 140-2 validated in several formats until September 1, 2020, when NIST ... designated all non-compliant modules as 'Historical'. This designation includes a caution to federal agencies that they should not include the module in any new procurements." Aside from the distros listed in the survey above which provide FIPS support, I do not think we need to go out of our way to test this any longer.

The Ubuntu 24.04 distro is selected for the openssl-compat matrix primarily to avoid complications with registering trusted certificates with the custom OpenSSL installation on RHEL or Arch Linux (no special handling seems to be required on Ubuntu). RHEL is also complicated due to backports of EVP_KDF_* functions into their OpenSSL 1.1.1 package and depending on it (e.g. by libk5crypto.so.3). Since the goal of this matrix is OpenSSL version compatibility checks, platform-specific OpenSSL compatibility is deferred to other existing -openssl tasks.

The openssl-compat matrix tasks also call the run-auth-tests.sh script via the run auth tests EVG command to extend compatibility checks to runtime behavior, not just compile-time compatibility as is the case for build-and-test-with-toolchain. This should hopefully allow us to detect issues like CDRIVER-5998 a bit better than before.

Numerous fixes and improvements are made to authentication-related tasks and scripts:

CDRIVER-5990: re-enable auth tests which were being skipped pending DEVPROD-16954, which was resolved in May 2025.
CDRIVER-5995: re-enable auth tests which were being skipped pending DEVPROD-9029, which was resolved in June 2025.
Fix authentication-* and ocsp-* task failures due to updated CMake binary directory paths.
- Update test-aws-* scripts for consistency.
Harden Bash scripts to ensure expanded variables are defined as expected.
- The atlas_x509_* Evergreen project variables required by run-auth-tests.sh have been moved into the EVG project repository settings so they are inherited by the r2.1 release branch.
Disabled several MacOS (darwinssl) OCSP tasks:
- MacOS ARM64 only provides MongoDB Server 6.0 or newer. Filtered pre-6.0 versions from the OCSP matrix accordingly.
  - Avoid extending coverage to MacOS Intel for pre-6.0 coverage, as these distro are very limited resources.
- mongoc does not seem to implement support for OCSP soft failure with the Secure Transport library (darwinssl). Does documentation need to be updated per spec? The tlsDisableOCSPEndpointCheck URI option is rejected by both Secure Transport (darwinssl) and Secure Channel (winssl), but neither of these behaviors seems to be publically documented.
- OCSP Test 3 also consistently fails with the Secure Transport library (darwinssl) despite Test 1 consistently passing, yet they "should be identical" per spec when OCSP stabling is not supported (which I don't think is supported for Secure Transport...?). I've disabled this task for now, but I don't think it should be left disabled.

eramongodb · 2025-08-11T15:40:57Z

CDRIVER-5990 unskip auth tests on zSeries

Per this patch build, zSeries no-SASL succeeds, but zSeries SASL fails during the "Authenticating using GSSAPI" test with:

ERROR:      cluster: Authentication failed: SASL Failure: (-1): generic failure
Ping failure: SASL Failure: (-1): generic failure

kevinAlbs

The script clean-ups and test coverage improvements are much appreciated.

The tlsDisableOCSPEndpointCheck URI option is rejected by both Secure Transport (darwinssl) and Secure Channel (winssl), but neither of these behaviors seems to be publically documented.

Later sections on Configure Transport Layer Security (TLS) note:

Setting tlsDisableOCSPEndpointCheck has no effect.

kevinAlbs · 2025-08-12T15:21:42Z

.evergreen/scripts/run-auth-tests.sh


 if [[ "${sasl}" != "OFF" ]]; then
  echo "Authenticating using GSSAPI"
-  LD_LIBRARY_PATH="${openssl_lib_prefix}" maybe_skip "${ping}" "mongodb://${auth_gssapi:?}@${auth_host}/?authMechanism=GSSAPI&${c_timeout}"
+  "${mongoc_ping:?}" "mongodb://${auth_gssapi:?}@${auth_host:?}/?authMechanism=GSSAPI&${c_timeout:?}"


CDRIVER-5990 unskip auth tests on zSeries

Per this patch build, zSeries no-SASL succeeds, but zSeries SASL fails during the "Authenticating using GSSAPI" test with:

ERROR: cluster: Authentication failed: SASL Failure: (-1): generic failure Ping failure: SASL Failure: (-1): generic failure

I would rather keep the maybe_skip on for zSeries, and leave CDRIVER-5990 unresolved for a further look. I suspect the failure environment related (I don't think GSSAPI code has changed recently).

Reverted as suggested.

kevinAlbs · 2025-08-12T15:27:29Z

.evergreen/scripts/openssl-downloader.sh

+# Download the requested OpenSSL version into `openssl-<version>`.
+openssl_download() {
+  declare version
+  version="${1:?"usage: openssl_download_checksum <version>"}"


Suggested change

version="${1:?"usage: openssl_download_checksum <version>"}"

version="${1:?"usage: openssl_download <version>"}"

kevinAlbs · 2025-08-12T15:40:22Z

.evergreen/scripts/run-ocsp-test.sh

+    if [[ -d "${openssl_install_dir:?}/lib64" ]]; then
+        LD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${LD_LIBRARY_PATH:-}"
+        DYLD_LIBRARY_PATH="${mongoc_build_dir:?}/src/libmongoc"
+    else
+        LD_LIBRARY_PATH="${openssl_install_dir:?}/lib:${LD_LIBRARY_PATH:-}"
+        DYLD_LIBRARY_PATH="${mongoc_build_dir:?}/src/libbson"
+    fi


Fix DYLD_LIBRARY_PATH overwrites:

Suggested change

if [[ -d "${openssl_install_dir:?}/lib64" ]]; then

LD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${LD_LIBRARY_PATH:-}"

DYLD_LIBRARY_PATH="${mongoc_build_dir:?}/src/libmongoc"

else

LD_LIBRARY_PATH="${openssl_install_dir:?}/lib:${LD_LIBRARY_PATH:-}"

DYLD_LIBRARY_PATH="${mongoc_build_dir:?}/src/libbson"

fi

if [[ -d "${openssl_install_dir:?}/lib64" ]]; then

LD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${LD_LIBRARY_PATH:-}"

DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${DYLD_LIBRARY_PATH:-}"

else

LD_LIBRARY_PATH="${openssl_install_dir:?}/lib:${LD_LIBRARY_PATH:-}"

DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib:${DYLD_LIBRARY_PATH:-}"

fi

kevinAlbs · 2025-08-12T15:56:19Z

.evergreen/scripts/openssl-compat-check.sh

+CMAKE_BUILD_PARALLEL_LEVEL="$(nproc)"
+
+# libmongocrypt must use the same OpenSSL library.
+echo "Installing libmongocrypt..."


Minor: Consider reusing compile-libmongocrypt.sh and passing the extra openssl_cmake_flags as extra arguments to ``compile_libmongocrypt` to avoid needing to update another location when updating libmongocrypt.

Done. Extended compile-libmongocrypt.sh to support additional arguments which are treated as extra CMake flags.

kevinAlbs · 2025-08-12T15:59:50Z

.evergreen/legacy_config_generator/evergreen_config_lib/tasks.py

+            # Why does this fail with Secure Transport (CSSMERR_TP_CERT_SUSPENDED)...?
+            prohibit(self.test == "TEST_3")
+
+            # Secure Transport (mongoc-stream-tls-secure-transport.c:_verify_peer) does not implement soft failure.


Suggested change

# Secure Transport (mongoc-stream-tls-secure-transport.c:_verify_peer) does not implement soft failure.

# TODO(CDRIVER-3759) Secure Transport fails soft failure test.

IIRC "soft-fail" behavior was only assumed after manual testing, and was assumed to be enabled with kSecRevocationOCSPMethod. Quoting https://mongodbcr.appspot.com/547790001/:

I can at least validate it is contacted (and then a soft-fail occurs)

But maybe the test failures suggest otherwise. I think investigating can be done as part of CDRIVER-3759.

This reverts commit ad45ce9.

kevinAlbs

LGTM with a typo fix.

kevinAlbs · 2025-08-12T20:10:08Z

.evergreen/scripts/run-ocsp-test.sh

+if [[ -d "${openssl_install_dir:?}" ]]; then
+    if [[ -d "${openssl_install_dir:?}/lib64" ]]; then
+        LD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${LD_LIBRARY_PATH:-}"
+        DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${LD_LIBRARY_PATH:-}"


Suggested change

DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${LD_LIBRARY_PATH:-}"

DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${DYLD_LIBRARY_PATH:-}"

kevinAlbs · 2025-08-12T20:10:16Z

.evergreen/scripts/run-ocsp-test.sh

+        DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${LD_LIBRARY_PATH:-}"
+    else
+        LD_LIBRARY_PATH="${openssl_install_dir:?}/lib:${LD_LIBRARY_PATH:-}"
+        DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib:${LD_LIBRARY_PATH:-}"


Suggested change

DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib:${LD_LIBRARY_PATH:-}"

DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib:${DYLD_LIBRARY_PATH:-}"

eramongodb added 8 commits August 6, 2025 12:15

Remove build-and-test-with-toolchain task

c42773d

Add openssl-downloader.sh

26a40be

CDRIVER-5995: unskip relevant auth tests

276acc1

Fix and update run-auth-tests.sh

385cbe2

Fix and update run-aws-tests.sh

1857626

Fix and update run-ocsp-test.sh

12ab0ce

Migrate OpenSSL compatibility checks into openssl-compat matrix

edd1d49

Disable failing OCSP darwinssl tasks

34761a0

eramongodb requested a review from kevinAlbs August 6, 2025 17:25

eramongodb self-assigned this Aug 6, 2025

eramongodb requested a review from a team as a code owner August 6, 2025 17:25

eramongodb added 3 commits August 6, 2025 12:39

Add verification of shared vs. static linkage of OpenSSL

3d3e177

Remove obsolete BYPASS_FIND_CMAKE branches

0ebb087

CDRIVER-5990 unskip auth tests on zSeries

ad45ce9

kevinAlbs reviewed Aug 12, 2025

View reviewed changes

eramongodb added 8 commits August 12, 2025 14:35

Revert "CDRIVER-5990 unskip auth tests on zSeries"

d59d135

This reverts commit ad45ce9.

Fix usage message for openssl_download

fdb8389

Extend openssl install paths to DYLD_LIBRARY_PATH

5ec26b7

Support forwarding extra CMake flags to compile_libmongocrypt

ee43567

Add reference to CDRIVER-3759

aab856a

Fix typo in variable expansion

4076371

Quote variable expansions

87bb0fe

Fix indexes after applying shifts

acc5524

eramongodb requested a review from kevinAlbs August 12, 2025 20:07

kevinAlbs approved these changes Aug 12, 2025

View reviewed changes

Fix copy-paste error for DYLD_LIBRARY_PATH

a1d0fd0

eramongodb merged commit 1f756d6 into mongodb:master Aug 12, 2025
35 of 36 checks passed

eramongodb deleted the cdriver-6071 branch August 12, 2025 20:30

eramongodb mentioned this pull request Aug 12, 2025

CDRIVER-6071 fix prohibition of OCSP TEST_3 with darwinssl #2086

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CDRIVER-6071 replace build-and-test-with-toolchain with openssl-compat #2082

CDRIVER-6071 replace build-and-test-with-toolchain with openssl-compat #2082

Uh oh!

eramongodb commented Aug 6, 2025 •

edited

Loading

Uh oh!

eramongodb commented Aug 11, 2025 •

edited

Loading

Uh oh!

kevinAlbs left a comment

Uh oh!

kevinAlbs Aug 12, 2025

Uh oh!

eramongodb Aug 12, 2025

Uh oh!

kevinAlbs Aug 12, 2025

Uh oh!

kevinAlbs Aug 12, 2025

Uh oh!

kevinAlbs Aug 12, 2025

Uh oh!

eramongodb Aug 12, 2025

Uh oh!

kevinAlbs Aug 12, 2025

Uh oh!

kevinAlbs left a comment

Uh oh!

kevinAlbs Aug 12, 2025

Uh oh!

kevinAlbs Aug 12, 2025

Uh oh!

Uh oh!

Uh oh!

	version="${1:?"usage: openssl_download_checksum <version>"}"
	version="${1:?"usage: openssl_download <version>"}"

	# Secure Transport (mongoc-stream-tls-secure-transport.c:_verify_peer) does not implement soft failure.
	# TODO(CDRIVER-3759) Secure Transport fails soft failure test.

	DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${LD_LIBRARY_PATH:-}"
	DYLD_LIBRARY_PATH="${openssl_install_dir:?}/lib64:${DYLD_LIBRARY_PATH:-}"

CDRIVER-6071 replace build-and-test-with-toolchain with openssl-compat #2082

CDRIVER-6071 replace build-and-test-with-toolchain with openssl-compat #2082

Uh oh!

Conversation

eramongodb commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eramongodb commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinAlbs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinAlbs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eramongodb commented Aug 6, 2025 •

edited

Loading

eramongodb commented Aug 11, 2025 •

edited

Loading